feat(sync): rsync-over-ssh NAS transport - #31
Open
Xander-git wants to merge 21 commits into
Open
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n test Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ode rclone bypass Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…arg default Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n rsync transport Argument-injection hardening from automated security review: every rsync argv now passes `--` before its positional local/remote arguments so a malformed config value beginning with `-` can never be parsed as a flag, and NasConfig rejects rsync-mode remotes whose user or host segment starts with `-`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Add rsync to the SFTP container Dockerfile (Debian rsync 3.2.7) - Provision a key-auth rsync user (rsyncuser) in entrypoint.sh: generates ed25519 keypair into bind-mounted keys/ dir on first boot, installs pubkey as authorized_keys, Match User block disables SFTP for that user while allowing rsync --server (Synology posture) - Set TZ=America/New_York on the nas-sftp service in docker-compose.yml so the container's timezone differs from the client's; this is the timezone characterization condition for the test suite - Drop --no-h from RsyncSshDriver's --list-only commands: the flag is GNU rsync-specific and fails on macOS openrsync (the client); the parser already handles comma-grouped sizes so --no-h was never needed - Add ssh_extra_opts test seam to RsyncSshDriver (keyword-only tuple, default empty) for injecting UserKnownHostsFile/StrictHostKeyChecking in integration tests without mutating developer's ~/.ssh/known_hosts; production config never sets it; unit test confirms argv rendering - Create tests/integration/test_rsync_ssh_characterization.py with docker-gated fixtures and 4 characterization tests: test_push_list_reconcile_roundtrip (TZ verdict: PASSED — timestamps are CLIENT-TZ formatted, spec assumption confirmed), test_check_differ_and_missing, test_list_only_on_missing_run_dir_raises_network, test_about_ok_and_missing_base_root - Document new user/keys/TZ in tests/docker/README.md Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Create docs/setup/rsync-ssh-setup.md: keypair generation (ed25519, empty passphrase), installing pubkey on Synology service account, host-key pre-provisioning via ssh-keyscan with explicit note that no login permission is needed (host-key exchange precedes auth), out-of-band fingerprint verification against DSM, recording rsync --version in the runbook, config.yaml block (rsync_ssh example from spec), degraded Test-connection note, and BatchMode failure-modes table - Add cross-link from docs/setup/rclone-remote-setup.md to rsync-ssh-setup.md - docs/design_specs/design_spec_sections/09_Configuration_File.md: add nas.transport / ssh_port / ssh_identity_file fields to the nas: block with inline comments - docs/design_specs/design_spec_sections/07_Sync_and_Database_Integration.md: add §7.1.3 transport matrix (rclone vs rsync_ssh) + verify-authority-differs- by-transport paragraph; update §7.1.3 header from "sole transport" to multi-transport framing - docs/design_specs/design_spec_sections/04_Backend_Architecture.md: add sync/transports/ subtree to the package layout; add NasTransportDriver Protocol + build_nas_driver factory paragraph - README.md: add one-line rsync-over-ssh transport mention + link to rsync-ssh-setup.md in the NAS sync setup section Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, exact stub dry-run match
Final-review findings: (1) the bare 'permission denied' auth marker
would terminally fail a job on a remote per-file filesystem error
('Permission denied (13)'); anchor on ssh's parenthesized auth-method
form instead. (2) An '-h'-formatted listing size silently dropped the
file from the manifest, re-queuing it forever — entry-shaped lines with
unparseable sizes now log a warning. (3) stub_rsync's dry-run detection
substring-matched 'n' in any flag; exact-match the driver's spellings.
The errors-always-empty CheckResult contract difference is now
documented in _synthesize_check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a second NAS transport — rsync over ssh — selected per instance by a new
nas.transportconfig field (rclonedefault,rsync_ssh). Cluster machines can only reach the Synology NAS via the allowlistedrsync --serverssh channel (outbound SMB blocked, rclone-SFTP locked by IT, no remote command exec, no interactive login); this branch makes those instances first-class sync clients while lab PCs keep rclone-over-SMB untouched.docs/superpowers/specs/2026-06-10-rsync-ssh-nas-transport-design.md(all open questions resolved; implementation amendments recorded in the status header)docs/superpowers/plans/2026-06-10-rsync-ssh-nas-transport.mddocs/setup/rsync-ssh-setup.md(keypair,ssh-keyscanhost-key provisioning — needs no login permission — cluster config block)How it works
RsyncSshDriver(sync/transports/rsync_ssh.py) behind a sharedNasTransportDriverprotocol +build_nas_driverfactory. Targets reuse the existing<remote>:<path>string (user@host:/pathis valid rsync syntax), so the queue state machine, reconcile loop, and cleanup gate are untouched.rsync -rt(mtime preservation contract) · manifest =--list-onlyparsed intoRemoteManifest(timestamp-anchored regex; spaces/escapes/comma-grouped sizes handled) · pre-deletion verify =--checksumdry-run itemize (remote-side hashing rides the same allowlisted channel) · probe = non-recursive listing (free-space readout degrades by design; no remote exec exists).BatchMode=yes, noStrictHostKeyCheckingrelaxation; host keys pre-provisioned viassh-keyscan.RcloneDriverregardless ofnas.transport(staging targets are rclone named remotes; the hidden staging backend keeps working) — pinned by a regression test.--before positionals; config rejects leading-dashuser/host; ssh auth-failure marker anchored onpermission denied (publickeyso a remote per-filePermission denied (13)stays retryable; listing parser warns loudly on-h-formatted sizes instead of silently dropping files.Both transports verified
test_nas_sync.pypassed 8/8 against the rebuilt sshd containerAmerica/New_York, which empirically confirmed--list-onlytimestamps are client-formattedDocker fixture grows a key-only rsync user (no SFTP subsystem — Synology posture) beside the untouched password-SFTP user.
Notes for reviewers
CheckResult.errorsis always()in rsync mode (itemize has no per-file error line; failures raise as classifiedTransportError) — documented contract difference.RsyncSshDriver.ssh_extra_optsis a documented test-only seam (integration fixture's throwawayknown_hosts); production config does not expose it.test_nas_client_stability.pyfails in isolation on a circular import that exists onmaintoo (passes in the full suite run).🤖 Generated with Claude Code